E2E: one-time global auth before test start by phyllis-sy-wu · Pull Request #7272 · Shopify/cli

phyllis-sy-wu · 2026-04-13T17:59:41Z

WHY are these changes introduced?

E2E tests currently authenticate the CLI and browser per worker on every test run. This PR centralizes authentication into a single globalSetup step that runs once before tests start.

This lays the foundation for:

Parallel workers (future PR): Without global auth, each worker would need to authenticate independently, causing redundant OAuth flows and potential rate limiting
Per-test store creation (future PR): Browser needs pre-authenticated cookies for admin.shopify.com to create dev stores via the store creation form

WHAT is this pull request doing?

Adds a Playwright globalSetup that authenticates once before tests start, then reuses the session:

CLI auth (setup/global-auth.ts): Spawns shopify auth login via PTY, completes OAuth in a headless browser (with passkey/WebAuthn bypass), waits for "Logged in"
Browser session establishment: Visits admin.shopify.com and dev.shopify.com to establish cookies for both domains (not just accounts.shopify.com)
Session reuse (setup/auth.ts): Copies the pre-authenticated CLI session files (XDG dirs) and loads browser storageState — no re-authentication needed

Also includes a CodeQL fix: URL checks for accounts.shopify.com redirects now use new URL().hostname comparison instead of substring matching.

Design decisions: session persistence

Approach: Follows Playwright's recommended storageState pattern — global setup authenticates once and saves browser cookies to a JSON file. Workers load the saved state into their browser context.

Always re-authenticate each run: Both local and CI use the same flow — full auth every run. This keeps the logic simple and consistent. No cross-run caching (on CI this is impossible since each run is a fresh container; locally the ~30s auth cost is acceptable).

Why per-worker copies of CLI config dirs? Browser auth shares one browser-storage-state.json fine — Playwright handles that natively. But CLI commands (app dev, app deploy) read and write to config directories at runtime (session tokens, cached app state). If parallel workers shared the same dirs, they'd corrupt each other's files. Each worker copies the auth artifacts into its own isolated XDG dirs.

Directory structure:

.e2e-tmp/
├── global-auth/                    ← stable dir, all auth artifacts together
│   ├── XDG_CONFIG_HOME/            ← CLI tokens (used on Linux CI)
│   ├── XDG_DATA_HOME/
│   ├── XDG_STATE_HOME/
│   ├── XDG_CACHE_HOME/
│   └── browser-storage-state.json  ← browser cookies (used everywhere)
├── e2e-{worker0}/                  ← per-worker isolated dirs
└── e2e-{worker1}/

Cross-platform note: On macOS, the CLI's conf package ignores XDG env vars and writes to ~/Library/Preferences/. The XDG dirs in global-auth/ are only populated on Linux. Browser storageState works on all platforms.

Files changed:

File	Change
`setup/global-auth.ts`	New: one-time auth + browser cookie establishment
`setup/auth.ts`	Copies session from global setup instead of re-authenticating
`setup/browser.ts`	Loads `storageState` from global setup
`setup/env.ts`	Adds `globalLog()` helper for pre-test logging
`playwright.config.ts`	Adds `globalSetup` entry

How to test your changes?

# Run all tests
DEBUG=1 pnpm --filter e2e exec playwright test

# Quick test with a single spec
DEBUG=1 pnpm --filter e2e exec playwright test app-deploy

# Watch browser behavior (headed mode)
E2E_HEADED=1 DEBUG=1 pnpm --filter e2e exec playwright test

# Expected output before tests start:
# [e2e][auth] global setup starting
# [e2e][auth] browser sessions established for admin + dev dashboard
# [e2e][auth] global setup done, config at ...

Checklist

I've considered possible cross-platform impacts (Mac, Linux, Windows)
I've considered possible documentation changes
I've considered analytics changes to measure impact
The change is user-facing, so I've added a changelog entry with pnpm changeset add

phyllis-sy-wu · 2026-04-13T17:59:56Z

Trial branch for more workers #7343
E2E: parallel workers #7309
E2E: per-test store isolation with automatic teardown #7279
E2E: temporary time limit bump #7282
E2E: improve error context in assertions #7281
E2E: one-time global auth before test start #7272 👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

phyllis-sy-wu

No issues found

Review assisted by pair-review

Copilot

Pull request overview

Introduces a Playwright globalSetup flow to authenticate the Shopify CLI and browser once per test run, then reuse the resulting CLI session files and browser storageState across worker-scoped fixtures.

Changes:

Added setup/global-auth.ts global setup to perform one-time CLI OAuth login and persist browser storage state.
Updated worker fixtures to reuse global session artifacts (copy XDG auth dirs + load storageState) instead of re-authenticating.
Added a globalLog() helper for pre-worker logging gated behind DEBUG=1.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
packages/e2e/setup/global-auth.ts	New global setup that performs one-time login and writes reusable session artifacts.
packages/e2e/setup/auth.ts	Reuses global CLI session by copying XDG dirs; retains fallback per-worker login.
packages/e2e/setup/browser.ts	Loads Playwright `storageState` from global setup when available.
packages/e2e/setup/env.ts	Adds `globalLog()` for debug logging during global setup.
packages/e2e/playwright.config.ts	Registers the new Playwright `globalSetup`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ryancbahan

The idea here is great! I'd encourage you to consider how other libraries and tools implement persistence between runs/manage sessions, though. This is a problem that has a lot of prior art to lean on. I don't think we'll end up needing multiple tmp dirs per-state. And I think some clarity in the pr description around choice of state prsistence and caching approach is important here.

…hopify.com

phyllis-sy-wu · 2026-04-16T22:12:55Z

@ryancbahan Thanks for the feedback!

I've updated the PR description with a "Design decisions" section covering the persistence approach. Please let me know if it’s still unclear.

Code changes based on this review:

Session caching: Added cross-run caching for local dev. Before re-authenticating, globalSetup loads the saved browser state and checks if it's still valid (~3s). If so, skips the full OAuth flow (~30s). I also explored doing the same on CI, but since CLI is a public repo, we can't safely cache auth tokens in GitHub Actions. So CI still re-authenticates fresh each run.
Consolidated auth dir: All auth artifacts now live in a stable global-auth/ directory under .e2e-tmp/ (instead of random mkdtemp dirs that accumulated across runs). Browser state JSON and XDG dirs are both inside it for a cleaner structure.

Things I kept:

Follows Playwright's recommended storageState pattern — global setup authenticates once, saves browser cookies to a JSON file, workers load it via storageState.
Per-worker temp dirs remain — workers copy auth into their own isolated XDG dirs because the CLI writes to config at runtime, so parallel workers need separate copies. This is the minimum for parallel execution.

More details in the updated PR description.

ryancbahan · 2026-04-17T02:01:06Z

      }

-      process.stdout.write('[e2e] Authenticating automatically — no action required.\n')
+      const authConfigDir = process.env.E2E_AUTH_CONFIG_DIR


do you need any of this at all if you run global auth once before the tests run, then don't call login() or any auth commands in any of the tests?

Yes! This copying step is needed for parallel worker isolation.

To be clear: this separation is for CLI command isolation, not for browser or login needs.

It’s correct that browser auth works fine with a single shared browser-storage-state.json — Playwright handles that natively. The problem is that our tests run CLI commands which read and write to config directories at runtime (storing app state, caching API responses, etc.). If multiple workers share the same config directory, they’d write to the same files simultaneously, causing race conditions and corrupted JSON.

.e2e-tmp/ ├── global-auth/ ← created once by globalSetup │ ├── XDG_CONFIG_HOME/ ← CLI tokens (source of truth) │ ├── XDG_DATA_HOME/ │ ├── XDG_STATE_HOME/ │ ├── XDG_CACHE_HOME/ │ └── browser-storage-state.json ← shared read-only by all workers ✅ │ ├── e2e-worker0/ ← worker 0's isolated copy │ ├── XDG_CONFIG_HOME/ ← copied from global-auth, written to by CLI commands │ ├── XDG_DATA_HOME/ ← worker 0's app deploy writes here │ └── ... │ └── e2e-worker1/ ← worker 1's isolated copy ├── XDG_CONFIG_HOME/ ← copied from global-auth, written to by CLI commands ├── XDG_DATA_HOME/ ← worker 1's app dev writes here └── ...

Global auth creates one set of CLI tokens in global-auth/. Each worker copies those into its own isolated XDG dirs so parallel CLI commands don’t interfere with each other.

The CLI writes to these local stores during commands:

Store Written during Conflict risk if shared

shopify-cli-kit Many commands (session tokens, current session ID, cached responses) HIGH — global fields like currentSessionId get overwritten; last write wins

shopify-cli-app app dev, app deploy (cached app info: appId, storeFqdn) Medium — entries are keyed per app dir, but conf rewrites the whole JSON per write, so concurrent writes can still race at the filesystem level

shopify-cli-store Store operations Medium — same file-level race risk as above

The critical one is cliKitStore — it stores the current session ID in a single field. If worker 0 and worker 1 write at the same time, one overwrites the other, causing auth failures.

Another original reason to keep auth.ts was a fallback path (authenticating directly without global setup) for local dev when running a single test without globalSetup. But both auth.ts and global-auth.ts use the same login logic — if global-auth fails, the fallback in auth.ts would fail the same way. So the fallback is effectively dead code on this branch. Only the copy step is strictly needed, and we could converge auth.ts and global-auth.ts in a follow-up PR. Does that make sense?

ryancbahan

I think it's worth exploring whether you can auth once per job run at the top level and pass the session in-memory. There's two reasons I think this approach merits consideration:

Storing on filesystem won't be useful in the GH actions runs since we spawn a new container each time
writing to disk means we need to pay more attention to if those files get exposed as build artifacts (which we don't want).

phyllis-sy-wu · 2026-04-17T03:59:44Z

I think it's worth exploring whether you can auth once per job run at the top level and pass the session in-memory.

Playwright communicates between globalSetup and workers via process.env + filesystem — there’s no built-in in-memory mechanism across worker processes. The CLI also reads tokens from disk (via the conf package), not memory, so filesystem is required for CLI commands to work.

Storing on filesystem won't be useful in the GH actions runs since we spawn a new container each time

You're right that cross-run caching doesn't help (fresh container). Within a single run, filesystem is still needed for Playwright's storageState and CLI token access. The caching is purely a local-dev optimization so that iteration locally can be quicker.

writing to disk means we need to pay more attention to if those files get exposed as build artifacts (which we don't want).

Auth files live in .e2e-tmp/ (gitignored). CI only uploads playwright-report/ and test-results/ — tokens are not exposed.

More details on why per-worker copies are needed in my reply to the auth.ts comment above.

gonzaloriestra

Nice work, thanks! I think the approach makes sense. And it's important to prepare for parallelization, because E2E is by far the slowest check right now.

gonzaloriestra · 2026-04-17T09:33:12Z

+    SHOPIFY_FLAG_CLIENT_ID: undefined,
+  }
+
+  // Check if cached session from a previous run is still valid


I think we don't need a cache only for local runs. It will be used very sporadically, and I think it's better to keep the same logic as CI. And code will be simpler as well.

phyllis-sy-wu · 2026-04-17T14:15:01Z

@ryancbahan @gonzaloriestra Thanks for the feedback!

After reconsidering, I agree that keeping the same behaviour for local and CI is simpler and easier to reason about. I've removed the local caching logic so both environments always re-authenticate fresh each run.

github-advanced-security AI found potential problems Apr 13, 2026

View reviewed changes

Comment thread packages/e2e/setup/global-auth.ts Fixed

Comment thread packages/e2e/setup/global-auth.ts Fixed

phyllis-sy-wu commented Apr 13, 2026

View reviewed changes

phyllis-sy-wu force-pushed the psyw-0413-E2E-one-time-auth branch from 76afe7f to 5f5c9e9 Compare April 13, 2026 18:32

phyllis-sy-wu marked this pull request as ready for review April 13, 2026 18:57

phyllis-sy-wu requested a review from a team as a code owner April 13, 2026 18:57

Copilot AI review requested due to automatic review settings April 13, 2026 18:57

Copilot started reviewing on behalf of phyllis-sy-wu April 13, 2026 18:57 View session

Copilot AI reviewed Apr 13, 2026

View reviewed changes

phyllis-sy-wu requested a review from ryancbahan April 13, 2026 19:59

This was referenced Apr 13, 2026

E2E: per-test store isolation with automatic teardown #7279

Open

E2E: improve error context in assertions #7281

Open

E2E: temporary time limit bump #7282

Open

phyllis-sy-wu force-pushed the psyw-0413-E2E-one-time-auth branch from d613bd8 to 3c8cf0e Compare April 15, 2026 03:26

phyllis-sy-wu mentioned this pull request Apr 15, 2026

E2E: parallel workers #7309

Open

4 tasks

ryancbahan requested changes Apr 16, 2026

View reviewed changes

phyllis-sy-wu added 6 commits April 16, 2026 12:48

E2E: one-time global auth before test start

e3c0788

fix: use URL hostname check instead of substring match for accounts.s…

097b81a

…hopify.com

replace custom copyDirSync with fs.cpSync

2e10289

remove redundant blank space

191ec88

use the same directories.root to keep temp files in one place

54c055d

fix: validate global auth artifacts and ensure resource cleanup

0e57009

phyllis-sy-wu force-pushed the psyw-0413-E2E-one-time-auth branch from 3c8cf0e to 121f09a Compare April 16, 2026 21:18

session caching + consolidate auth dir

cb219bf

phyllis-sy-wu force-pushed the psyw-0413-E2E-one-time-auth branch from 121f09a to cb219bf Compare April 16, 2026 22:11

phyllis-sy-wu requested a review from ryancbahan April 17, 2026 00:20

ryancbahan reviewed Apr 17, 2026

View reviewed changes

gonzaloriestra approved these changes Apr 17, 2026

View reviewed changes

remove session caching for local dev

698c2b4

phyllis-sy-wu force-pushed the psyw-0413-E2E-one-time-auth branch from 73f8d68 to 698c2b4 Compare April 17, 2026 14:13

phyllis-sy-wu requested a review from ryancbahan April 17, 2026 14:15

phyllis-sy-wu mentioned this pull request Apr 17, 2026

Trial branch for more workers #7343

Draft

4 tasks

Store	Written during	Conflict risk if shared
`shopify-cli-kit`	Many commands (session tokens, current session ID, cached responses)	HIGH — global fields like `currentSessionId` get overwritten; last write wins
`shopify-cli-app`	`app dev`, `app deploy` (cached app info: appId, storeFqdn)	Medium — entries are keyed per app dir, but `conf` rewrites the whole JSON per write, so concurrent writes can still race at the filesystem level
`shopify-cli-store`	Store operations	Medium — same file-level race risk as above

Conversation

phyllis-sy-wu commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

WHY are these changes introduced?

WHAT is this pull request doing?

Design decisions: session persistence

How to test your changes?

Checklist

Uh oh!

phyllis-sy-wu commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phyllis-sy-wu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryancbahan left a comment

Choose a reason for hiding this comment

Uh oh!

phyllis-sy-wu commented Apr 16, 2026

Uh oh!

ryancbahan Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

phyllis-sy-wu Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

ryancbahan left a comment

Choose a reason for hiding this comment

Uh oh!

phyllis-sy-wu commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gonzaloriestra left a comment

Choose a reason for hiding this comment

Uh oh!

gonzaloriestra Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

ryancbahan Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

phyllis-sy-wu commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

phyllis-sy-wu commented Apr 13, 2026 •

edited

Loading

phyllis-sy-wu commented Apr 13, 2026 •

edited

Loading

phyllis-sy-wu left a comment •

edited

Loading

phyllis-sy-wu commented Apr 17, 2026 •

edited

Loading

phyllis-sy-wu commented Apr 17, 2026 •

edited

Loading